swift_too module

Swift_Data example - querying Swift SDC data and downloading it

API version = 1.2, swifttools = 3.0.6

Author: Jamie A. Kennea (Penn State)

from swifttools.swift_too import Data
from swifttools.swift_too import ObsQuery

Introduction

We've demonstrated that with classes such as Swift_ObsQuery we can find out what Swift has observed, but what if we want to download the data associated with an observation? Before you had to go to a web interface to download the data you wanted. More confusingly, the data usually reside in one of two places: the archive or the quicklook area. Typically recent observations exist in quicklook, and older observations are in the archive.

Finding an interesting observation

Swift_Data aims to make fetching observation data automated, and somewhat abstracted, so you don't have to care about if data are from quicklook or not. Let's find a recent observation, and download it. How about the first few observations of Jan 10th, 2021? As of swifttools 2.3, you can shorten the names of the classes by removing the Swift_ so we'll use just Data and for the next example, ObsQuery.

oq = ObsQuery(begin="2022-01-10", length=0.1)
oq

OK, so that observation fo NGC 5986 is interesting, so let's check it out. The important identifier for any observation is the Observation Number (or Observation ID). In the table above, this is given in SDC format, which is a importantly, a string (those leading zeros are required). So for this observation, the obsid is 00045907009. However, we don't have to make transcription errors here, it's the first in the oq data structure, i.e...

oq[0]

Demonstrating the Swift_Data class

So let's demonstrate use of the Swift_Data class. As usual, it can be set up by creating the class and then setting the parameters, then submitting using the submit method, or you can pass the parameters as arguments to the class. We'll do the former first.

data = Data()
data.obsid = oq[0].obsid

OK so this sets up that we want to fetch data associated with the obsid for NGC5986. Next we have a few options to set. Firstly, there are 5 types of data that can be downloaded. These are:

• auxil : Auxillary data required to process Swift data, including Two Line Element (TLE) to calculated an ephemeris, and spacecraft housekeeping.
• log : Log files from the SDC processing. Of interest, but usually inessential for data analysis.
• xrt : Data for Swift's X-ray Telescope.
• bat : Data for Swift's Burst Alert Telescope
• uvot : Data for Swift's UV/Optical Telescope

In order to select which data you wish to download, simply set the parameter listed above to true, so if you want XRT data, set xrt = True. If you just want to get everything, you can set all = True as a shortcut. Let's get XRT, BAT and UVOT data for this observation, forget about the logs. FYI auxil data come down by default, as you always need those if you're going to run processing pipeline tools like xrtpipeline.

data.xrt = True
data.uvot = True
data.bat = True

So in terms of basic configuration, this is pretty much it. Next step is to query the existance of data. This is done by submitting the job.

data.submit()

True

OK that appeared to have worked, so let's look at the files available. In a Jupyter Notebook, just type in the name of the Swift_Data object we created.

data

OK so what you should see above is a list of all the files, including the subdirectories they will exist in, along with basic description of each file. Importantly, at this stage, the data are not yet downloaded. Let's look in depth at one of these files.

data[0]

OK so a few things here. Firstly, from the url you can see where the data reside online. As this is a nasa.gov address, this is the US data center. If you wanted to retrieve from the UK data center, simply pass the parameter uksdc = True. Similarly if you wish to use the Italian SDC, use itsdc = True.

Secondly, this is quicklook data. By default Swift_Data looks in the archive first, but if the data is recent, it will look in quicklook. Note that quicklook data are by their nature preliminary, and can evolve with time. For US and Italian SDC you'll see that the URL has a version number ('009') included, this number increments when the data are changed. For UK SDC, this does not happen, however, you can just redownload data to get the most up-to-date value.

For published papers, we recommend using data from the archive if available.

Downloading the data

Final step is to download the data to disk, so we can process the data using our HEAsoft tools. This is perfomed utilizing the download() method. If we want to specify the directory to download, simply use the outdir parameter, either as a class parameter, or an argument to download. Let's put in in our computer's Downloads directory.

data.download(outdir="~/Downloads")

Downloading files:   0%|          | 0/57 [00:00<?, ?files/s]

True

If you have tqdm installed, you should have seen a progress bar showing how many files were downloaded. If not, then you'll just see a simple text message. Now the data should be in the Downloads directory, in a directory named after the obsid of the data we requested. Note that the files inside are gzipped by default, but do not need to be decompressed to be used with the HEAsoft tools, so you can save some disk space!

Download data in one line

Next thing we'll demonstrate is downloading data in one line of code. This is very simple, just pass the arguments when you intiate the class, and it will download the data itself. If you want to do this, and not download the data automatically, pass the argument fetch=False. We'll fetch the 3rd observation above, but just the XRT data:

newdata = Data(obsid=oq[2].obsid, xrt=True, outdir="~/Downloads")

Downloading files:   0%|          | 0/27 [00:00<?, ?files/s]

That should have downloaded the auxil and xrt data for that observation in into your Downloads directory. Let's look at the file structure.

newdata

Downloading data without using the Data class

You don't actually need to use the Swift_Data AKA Data class to download data. Any API class that has an associated obsid as of swifttools 3.0.6 has a download() method. This method takes the same arguments as Data to download data, automatically downloads data to local disk, and returns a Data class as a result. Here's a demonstration:

A reminder, we did an ObsQuery earlier, let's look at that again:

oq

The 5th observation up there is Mrk 817, we can access that entry individually like this:

oq[4]

So to download the data for that observation, you can simply do the following. We're going to request XRT and UVOT data for this:

oq[4].download(xrt=True,uvot=True, outdir="~/Downloads")

Downloading files:   0%|          | 0/60 [00:00<?, ?files/s]

That makes downloading data for archival observations quick and easy!

Updating data, fetching additional instruments

OK, so what if you want to update directory, say the data has been updated. Well we would recommend that you delete the old data, and redownload, and to encourage this, if you try to download again, you'll get an error:

newdata = oq[2].download(xrt=True, outdir="~/Downloads")

Downloading files:   0%|          | 0/27 [00:00<?, ?files/s]

WARNING: SWIFT_TLE_ARCHIVE.txt.22020.02942007.gz exists and not overwritten (set clobber=True to override this).
WARNING: sw01091527009pat.fits.gz exists and not overwritten (set clobber=True to override this).
WARNING: sw01091527009pjb.par.gz exists and not overwritten (set clobber=True to override this).
WARNING: sw01091527009pob.cat.gz exists and not overwritten (set clobber=True to override this).
WARNING: sw01091527009ppr.par.gz exists and not overwritten (set clobber=True to override this).
WARNING: sw01091527009s.mkf.gz exists and not overwritten (set clobber=True to override this).
WARNING: sw01091527009sao.fits.gz exists and not overwritten (set clobber=True to override this).
WARNING: sw01091527009sat.fits.gz exists and not overwritten (set clobber=True to override this).
WARNING: sw01091527009sen.hk.gz exists and not overwritten (set clobber=True to override this).
WARNING: sw01091527009sti.fits.gz exists and not overwritten (set clobber=True to override this).
WARNING: sw01091527009uat.fits.gz exists and not overwritten (set clobber=True to override this).
WARNING: sw01091527009x.mkf.gz exists and not overwritten (set clobber=True to override this).
WARNING: sw01091527009xpcw3po_cl.evt.gz exists and not overwritten (set clobber=True to override this).
WARNING: sw01091527009xpcw3po_uf.evt.gz exists and not overwritten (set clobber=True to override this).
WARNING: sw01091527009xwtw2sl_cl.evt.gz exists and not overwritten (set clobber=True to override this).
WARNING: sw01091527009xwtw2sl_uf.evt.gz exists and not overwritten (set clobber=True to override this).
WARNING: sw01091527009xwtw2sl_ufre.evt.gz exists and not overwritten (set clobber=True to override this).
WARNING: sw01091527009xwtw2st_cl.evt.gz exists and not overwritten (set clobber=True to override this).
WARNING: sw01091527009xwtw2st_uf.evt.gz exists and not overwritten (set clobber=True to override this).
WARNING: sw01091527009xwtw2st_ufre.evt.gz exists and not overwritten (set clobber=True to override this).
WARNING: sw01091527009xbf_rw.img.gz exists and not overwritten (set clobber=True to override this).
WARNING: sw01091527009xen.hk.gz exists and not overwritten (set clobber=True to override this).
WARNING: sw01091527009xhd.hk.gz exists and not overwritten (set clobber=True to override this).
WARNING: sw01091527009xtr.hk.gz exists and not overwritten (set clobber=True to override this).
WARNING: sw01091527009x_skim.gif exists and not overwritten (set clobber=True to override this).
WARNING: sw01091527009xpc_ex.img.gz exists and not overwritten (set clobber=True to override this).
WARNING: sw01091527009xpc_sk.img.gz exists and not overwritten (set clobber=True to override this).

However, we can overwrite the existing data, by (as the error message suggests), setting clobber = True.

newdata.clobber = True
newdata.download()

Downloading files:   0%|          | 0/27 [00:00<?, ?files/s]

True

Use of clobber can be useful also if you decide to you really wanted that UVOT data to go along with your XRT data on that observation. Remember auxil is fetched by default, so as we already have it, we'll turn it off. This isn't necessary, but if you don't turn it off the data will be redownloaded and overwrite the old data, which wastes bandwidth.

newdata = obsid=oq[2].download(uvot=True, auxil=False, outdir="~/Downloads", clobber=True)

Downloading files:   0%|          | 0/17 [00:00<?, ?files/s]

So the looking in the directory listing will show the UVOT data files that you just downloaded to add to your XRT and Auxillary data.

Selecting individual files using filename matching

If you want to specify exactly what sort of data you want to download, then you can use pattern matching to filter on only the files you want. Here's how that works. Firstly, let's use a GUANO dump as an example, for these we are primarily interested in the BAT event data file.

from swift_too import GUANO

g = GUANO(begin='2021-01-01',limit=1)
g

Let's take a look at the data associated with this GUANO trigger. I'll use the download() method, but set fetch = False so that no actual data is downloaded, this will return the directory listing.

g[0].download(bat=True,fetch=False)

So we're interested in just fetching the BAT event file. That file is called sw00048753206bevshpo_uf.evt.gz in the listing above. To fetch this only, we could use the match argument, and do the following:

g[0].download(bat=True,fetch=False,match='*bev*.evt.gz')

Here we used standard unix/dos style wildcards to only select files that are BAT event files ('b' for BAT, 'ev' for event). However, we probably want those files in the auxil directory too, so let's make sure we get those also with our match, by adding a second match clause.

g[0].download(bat=True,fetch=False,match=['*bev*.evt.gz','*/auxil/*'])

That's pretty much all there is to it! Enjoy your data.